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Response time as defined for this study is the time that 
it takes for all files that constitute a single webpage to 
travel across the Internet from a Web server to the end 
user's browser. In this study, the authors tested response 
times on queries for identical items in five different 
library catalogs, one of them a next-generation (NextGen) 
catalog. The authors also discuss acceptable response time 
and how it may affect the discovery process. They suggest 
that librarians and vendors should develop standards for 
acceptable response time and use it in the product selec- 
tion and development processes. 

N ext-generation, or NextGen, library catalogs offer 
advanced features and functionality that facilitate 
library research and enable Web 2.0 features such 
as tagging and the ability for end users to create lists and 
add book reviews. In addition, individual catalog records 
now typically contain much more data than they did in 
earlier generations of online catalogs. This additional 
data can include the previously mentioned tags, lists, 
and reviews, but a bibliographic record may also con- 
tain cover images, multiple icons and graphics, tables of 
contents, holdings data, links to similar items, and much 
more. This additional data is designed to assist catalog 
users in the selection, evaluation, and access of library 
materials. However, all of the additional data and features 
have the disadvantage of increasing the time it takes for 
the information to flow across the Internet and reach the 
end user. Moreover, the code that handles all this data 
is much more complex than the coding used in earlier, 
traditional library catalogs. Slow response time has the 
potential to discourage both library patrons from using 
the catalog and library staff from using or recommending 
it. During a reference interview or library instruction ses- 
sion, a slow response time creates an awkward lull in the 
process, a delay that decreases confidence in the mind of 
library users, especially novices who are accustomed to 
the speed of an open Internet search. 

The two-fold purpose of this study is to define the 
concept of response time as it relates to both traditional 
and NextGen library catalogs and to measure some typical 
response times in a selection of library catalogs. Libraries 
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and librarians will benefit from knowing what typical and 
acceptable response times are in online catalogs, and this 
information will assist in the design and evaluation of 
library discovery systems. This study also looks at bench- 
marks in response time and defines what is unacceptable 
and why. When advanced features and content in library 
catalogs increase response time to the extent that users 
become disaffected and use the catalog less, NextGen cata- 
logs represent a step backward, not forward. 

In August 2009, the Auraria Library launched an 
instance of the WorldCat Local product from OCLC, 
dubbed WorldCat@Auraria. The Library's traditional 
catalog — named Skyline and running on the Innovative 
Interfaces platform — still runs concurrently with 
WorldCat@Auraria. Because WorldCat Local currently 
lacks a library circulation module that the Library was 
able to use, the legacy catalog is still required for its 
circulation functionality. In addition. Skyline contains 
MARC records from the SerialsSolution 360 MARC prod- 
uct. Since many of these records are not yet available in 
the OCLC WorldCat database, these records are being 
maintained in the legacy catalog to enable access to the 
Library's extensive collection of online journals. 

Almost immediately upon implementation of 
WorldCat Local, many Library staff began to express 
concern about the product's slow response time. They 
bemoaned its slowness both at the reference desk and 
during library instruction sessions. Few of the discus- 
sions of the product's slow response time evaluated 
this weakness in the context of its advanced features. 
Several of the reference and instruction librarians even 
stated that they refused to use it any longer and that 
they were not recommending it to students and faculty. 
Indeed, many stated that they would only use the legacy 
Skyline catalog from then on. Therefore we decided to 
analyze the product's response time in relation to the 
legacy catalog. We also decided to further our study by 
examining response time in library catalogs in general, 
including several different online catalog products from 
different vendors. 


■ Response Time 

The term response time can mean different things in dif- 
ferent contexts. Here we use it to mean the time it takes 
for all files that constitute a single webpage (in the case of 
testing performed, a permalink to a bibliographic record) 
to travel across the Internet from a Web server to the 
computer on which the page is to be displayed. We do 
not include the time it takes for the browser to render the 
page, only the time it takes for the files to arrive to the 
requesting computer. Typically, a single webpage is made 
of multiple files; these are sent via the Internet from a Web 
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server and arrive sequentially at the computer where the 
request was initiated. 

While the World Wide Web Consortium (W3C) does 
not set forth any particular guidelines regarding response 
time, go-to usability expert Jakob Nielsen states that "0.1 
second is about the limit for having the user feel that the 
system is reacting instantaneously." 1 He further posits 
that 1.0 second is "about the limit for the user's flow of 
thought to stay uninterrupted, even though the user will 
notice the delay." 2 Finally, he asserts that: 

10 seconds is about the limit for keeping the user's 
attention focused on the dialogue. For longer delays, 
users will want to perform other tasks while waiting 
for the computer to finish, so they should be given 
feedback indicating when the computer expects to be 
done. Feedback during the delay is especially impor- 
tant if the response time is likely to be highly variable, 
since users will then not know what to expect. 3 

Even though this advice dates to 1994, Nielsen noted 
even then that it had "been about the same for many 
years." 4 


■ Previous Studies 

The chief benefit of studying response time is to estab- 
lish it as a criterion for evaluating online products 
that libraries license and purchase, including NextGen 
online catalogs. Establishing response-time benchmarks 
will aid in the evaluation of these products and will 
help libraries convey the message to product vendors 
that fast response time is a valuable product feature. 
Long response times will indicate that a product is 
deficient and suffers from poor usability. It is important 
to note, however, that sometimes library technology 
environments can be at fault in lengthening response 
time as well; in "Playing Tag In the Dark: Diagnosing 
Slowness In Library Response Time," Brown-Sica diag- 
nosed delays in response time by testing such variables 
as vendor and proxy issues, hardware, bandwidth, and 
network traffic. 5 In that case, inadequate server specifi- 
cations and settings were at fault. 

While there are many articles on NextGen catalogs, 
few of them discuss the issue of response time in rela- 
tion to their success. Search slowness has been reported 
in library literature about NextGen catalogs' metasearch 
cousins, federated search products. In a 2006 review 
of federated search tools MetaLib and WebFeat, Chen 
noted that "a federated search could be dozens of times 
slower than Google." 6 More comments about the negative 
effects of slow response time in NextGen catalogs can be 
found in popular library technology blogs. On his blog. 


Mathews posted an article called "5 Next Gen Library 
Catalogs and 5 Students: Their Initial Impressions." 7 
Here he shares student impressions of several NextGen 
catalogs. Regarding slow response time Mathews notes, 
"Lots of comments on slowness. One student said it took 
more than ten seconds to provide results. Some other 
comments were: 'that's unacceptable' and 'slow-motion 
search, typical library.'" Nagy and Garrison, on Lauren's 
Library Blog, emphasized that any "cross-silo federated 
search" is "as slow as the slower silos." 8 Any search inter- 
face is as slow as the slowest database from which it pulls 
information; however, that does not make users more 
likely to wait for search results. In fact, many users will 
not even know — or care — what is happening behind the 
scenes in a NextGen catalog. 

The assertion that slow response time makes well- 
intentioned improvements to an interface irrelevant is 
supported by an article that analyzes the development of 
Semantic Web browsers. Frachtenberg notes that 

users, however, have grown to expect Web search 
engines to provide near-instantaneous results, and a 
slow search engine could be deemed unusable even 
if it provides highly relevant results. It is therefore 
imperative for any search engine to meet its users' 
interactivity expectations, or risk losing them. 9 

This is not just a library issue. Users expect a fast 
response to all Web queries, and we can learn from 
studies on general Web response time and how it 
affects the user experience. Huang and Fong-Ling help 
explain different user standards when using websites. 
Their research suggests that "hygiene factors" such as 
"navigation, information display, ease of learning and 
response time" are more important to people using 
"utilitarian" sites to accomplish tasks rather than "hedo- 
nistic" sites. 10 In other words, response time importance 
increases when the user is trying to perform a task — 
such as research — and possibly even more for a task that 
may be time sensitive — such as trying to complete an 
assignment for class. 


■ Method 

For testing response time in an assortment of library cat- 
alogs, we used the WebSitePulse service (http://www 
.websitepulse.com). WebSitePulse provides in-depth 
website and server diagnostic services that are intended to 
save e-business customers time and money by reporting 
errors and Web server and website performance issues to 
clients. A thirty-day free trial is available for potential cus- 
tomers to review the full array of their services; however, 
the free Web Page Test, available at http://www.website 
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Web Page Test results 

URL tested: http://skyline.cudenver.edu/record=b2433301~S0 

Test performed from: Seattle, WA 

Test performed at: 2010-02-23 14:49:10 (GMT -08:00) 



Time (seconds) 


I I DNS □ Connect B Redirect Q First byte Q Last byte Q Error 


Figure 1 . Permalink screen shot for the record for the title Hard 
Lessons in Auraria Library’s Skyline catalog 


Figure 2. WebSitePulse webpage test bar graph results for 
Skyline (traditional) catalog record 


URL tested: httD://skvline.cudenver.edu/record = 

Test performed from: Seattle, WA 

Test performed at: 2010-02-23 14:49:10 (GMT -08:00) 


pulse.com/corporate/alltools.php, 
met our needs. To use the webpage 
test, simply select "Web Page Test" 
from the dropdown menu, input 
a URL — in the case of the testing 
done for this study, the perma- 
link for one of three books (see, 
for example, figure 1) — enter the 
validation code, and click "Test It." 

WebSitePulse returns a bar graph 
(figure 2) and a table (figure 3) of 
the file activity from the server 
sending the composite files to the 
end user's Web browser. Each line 
represents one of the files that make 
up the rendered webpage. They 
load sequentially, and the bar graph 
shows both the time it took for each 
file to load and the order in which 
the files were received. Longer seg- 
ments of the bar graph provide 
visual indication of where a slow-loading webpage 
might encounter sticking points — for example, wait- 
ing for a large image file or third-party content to load. 
Accompanying the bar graph is a table describing the file 
transmissions in more detail, including DNS, connection, 
file redirects (if applicable), first and last bytes, file trans- 
mission times, and file sizes. 


Web Page Test Result 

Test performed by WebSitePulse on 23 February, 2010 and emailed to you by nina (nina.mchale@ucdenver.edu). 


b2433301~S0 



# URL 

Status 

Time 

DNS (sec) 

Connect (sec) Redirect (sec) 

First (sec) 

Last (sec) Total (sec) 

Size (Kb) 

1 httD: / / skyline. cudenver.edu / record -1)24 3330 

~S0 OK 

14:49:10 

0.0009 

0.0434 

0.0000 

0.0583 

0.0809 

0.1835 

12.22 

2 skyline.cudenver.edu/scnpts/iiiSryles.css 

OK 

14:49:10 

0.0000 

0.0000 

0.0000 

0.0494 

0.0004 

0.0499 

5.80 

3 skvline.cudenver.edu/screens/stvles.css 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0462 

0.0435 

0.0898 

10.41 

4 skyline.cudenver.edu/screens/favicon.ico 

404 - Not Found 14:49:11 

0.0000 

0.0000 

0.0000 

0.0448 

0.0002 

0.0451 

0.12 

S skyline.cudenver.edu/scripts/common.is 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0459 

0.0882 

0.1342 

36.77 

6 skvline.cudenver.edu/screens/backaround.ipa 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0461 

0.0001 

0.0462 

0.33 

7 skyline.cudenver.edu/screens/headercombo.ipa 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0487 

0.0001 

0.0488 

6.81 

8 skyline.cudenver.edu/screens/startover.qif 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0456 

0.0001 

0.0457 

1.81 

9 skvline.cudenver.edu/screens/reauest.aif 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0456 

0.0001 

0.0458 

1.99 

10 skyline.cudenver.edu/screens/bibexport.qif 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0457 

0.0000 

0.0458 

1.91 

11 skvline.cudenver.edu/screens/marcdisD.aif 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0455 

0.0000 

0.0456 

2.15 

12 skyline.cudenver.edu/screens/qrlinker.qif 

OK 

14:49:11 

0.0001 

0.0000 

0.0000 

0.0453 

0.0001 

0.0455 

2.27 

13 skyline.cudenver.edu/screens/another.qif 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0455 

0.0001 

0.0457 

2.01 

14 skyline.cudenver.edu/screens/spacer.qif 

OK 

14:49:11 

0.0000 

0.0000 

0.0000 

0.0455 

0.0001 

0.0456 

0.04 

Total 


- 

0.0010 

0.0434 

0.0000 

0.6581 

0.2139 

0.9172 

84.64 



Figure 3. WebSitePulse webpage test table results for Skyline (traditional) catalog record 


I 


Findings: Skyline Versus 
WorldCat@Auraria 


In figure 2, the bar graph shows a sample load time for 
the permalink to the bibliographic record for the title 
Hard Lessons: The Iraq Reconstruction Experience in Skyline, 
Auraria's traditional catalog load time for the page is 
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Figure4. Permalink screen shot for the record for the title Hard 
Lessons in WorldCat@Auraria 


1.1429 seconds total. The record is composed of a total 
of fourteen items, including image files (GIFs), cascad- 
ing style sheet (CSS) files, and JavaScript (JS) files. As the 
graph is read downward, the longer segments of the bars 
reveal the sticking points. In the case of Skyline, the nine 
image files, two CSS files, and one JS file loaded quickly; 
the only cause for concern is the red line at item four. This 
revealed that we were not taking advantage of the option 
to add a favicon to our III catalog. The Web librarian 
provided the ILS server technician with the same favi- 
con image used for the Library's website, correcting this 
issue. The Skyline catalog, judging by this data, falls into 
Nielsen's second range of user expectations regarding 
response time, which is more than one second, or "about 
the limit for the user's flow of thought to stay uninter- 
rupted, even though the user will notice the delay." 11 
Further detail is provided in figure 3; this table lists each 
of the webpage's component files, and various times asso- 
ciated with the delivery of each file. The column on the 
right lists the size in kilobytes of each file. The total size 
of the combined files is 84.64 KB. 

In contrast to Skyline's meager 14 files, WorldCat 
Local requires 31 items to assemble the webpage (figure 
4) for the same bibliographic record. Figures 5 and 6 
show that this includes 10 CSS files, 10 JavaScript files, 
and 8 images files (GIFs and PNGs). No item in particular 
slows down the overall process very much; the longest- 
loading item is number 13, which is a wait for third-party 
content, a connection to Yahool's User Interface (YUI) 
API service. Additional third-party content is being 


Web Page Test results 

URL tested: http://aurarialibrary.worldcat.org/odc/302189848 

Test performed from: Seattle, WA 

Test performed at: 2010-02-23 14:52:43 (GMT -08:00) 



Time (seconds) 


I I DNS Connect Q] Redirect First byte Last byte |3] Error 


Figure 5. WebSitePulse webpage test bar graph results for 
WorldCat@Auraria record 


requested at items 8, 14, 15, 17, 26, and 27. The third 
parties include Yahoo! API services, the Google API ser- 
vice, ReCAPTCHA, and AddThis. ReCAPTCHA is used 
to provide security within WorldCat Local with opti- 
cal character recognition images ("captchas"), and the 
AddThis API is used to provide bookmarking function- 
ality. At number 22, a connection is made to the Auraria 
Library Web server to retrieve a logo image hosted on 
the Web server. At number 28, the cover photo for Hard 
Lessons is retrieved from an OCLC server. The files listed 
in figure 6 details the complex process of Web brows- 
ers' assembly of them. Each connection to third-party 
content, while all relatively short, allows for addi- 
tional features and functionality, but lengthens overall 
response. As figure 6 shows, the response time is slightly 
more than 10 seconds, which, according to Nielsen, "is 
about the limit for keeping the user's attention focused 
on the dialogue." 12 While widgets, third-party content, 
and other Web 2.0 tools add desirable content and 
functionality to the Library's catalog, they also do slow 
response time considerably. The total file size for the 
bibliographic record in WorldCat@Auraria — compared 
to Skyline's 84.64 KB — is 633.09 KB. As will be shown 
in the test results below for the catalog and NextGen 
catalog products, bells and whistles added to traditional 
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total time for each permalinked 
bibliographic record to load as 
reported by the WebSitePulse tests; 
this number appears near the lower 
right-hand corner of the tables in 
figures 3, 6, 9, 12, and 15. 

We selected three books that 
were each held by all five of our test 
sites, verifying that we were search- 
ing the same three bibliographic 
records in each of the online catalogs 
by looking at the OCLC number in 
the records. Each of the catalogs we 
tested has a permalink feature; this 
is a stable URL that always points 
to the same record in each catalog. 
Using a permalink approximates 

for that item from a catalog search 
screen. We saved these links and 
used them in our searches. The bib- 
liographic records we tested were for 
Figure 6. WebSitePulse webpage test table results for WorldCat@Auraria record these books; the permalinks used for 

testing follow the books: 


conducting a known-item search 



catalogs slowed response time considerably, even dou- 
bling it in one case. Are they worth it? The response of 
Auraria's reference and instruction staff seems to indi- 
cate that they are not. 

I Gathering More Data: Selecting the 
Books and Catalogs to Study 

To broaden our comparison and to increase our data 
collection, we also tested three other non-Auraria cata- 
logs. We designed our study to incorporate a number of 
variables. We decided to link to bibliographic records for 
three different books in the five different online catalogs 
tested. These included Skyline and WorldCat@Auraria 
as well three additional online public access catalog 
products, for a total of two instances of Innovative 
Interfaces products, one of a Voyager catalog, and one 
of a SirsiDynix catalog. We also selected online catalogs 
in different parts of the country: WorldCatLocal in Ohio; 
Skyline in Denver; the Library of Congress' Online 
Catalog (LCOC) in Washington, D.C.; the University 
of Texas at Austin's (UT Austin) online catalog; and 
the University of Southern California's (USC) online 
catalog, named Homer, in Los Angeles. We also did our 
testing at different times of the day. One book was tested 
in the morning, one at midday, and one in the afternoon. 
WebSitePulse performs its webpage tests from three 
different locations in Seattle, Munich, and Brisbane; 
we selected Seattle for all of our tests. We recorded the 


Book 1: Hard Lessons: The Iraq Reconstruction Experience. 
Washington, D.C.: Special Inspector General, Iraq 
Reconstruction, 2009 (OCLC number 302189848). 

Permalinks used: 

■ WorldCat@Auraria: http://aurarialibrary.worldcat 
.org/ oclc/ 302189848 

■ Skyline: http://skyline.cudenver.edu/recordH3243 
3301 -SO 

■ LCOC: http://lccn.loc.gov/2009366172 

■ UT Austin: http: //catalog. lib. utexas.edu/record = 
b7195737~S29 

■ USC: http://library.usc.edu/uhtbin/cgisirsi/ 

x/0/ 0/5?searchdatal=2770895)CKEY) 

Book 2: Ehrenreich, Barbara. Nickel and Dimed: On (Not) 
Getting by in America. 1st ed. New York: Metropolitan, 
2001 (OCLC number 256770509). 

Permalinks used: 

■ WorldCat@Auraria: http://aurarialibrary.worldcat 
.org/ oclc/ 45243324 

■ Skyline: http://skyline.cudenver.edu/recordH3l87 
0305-S0 

■ LCOC: http:/ /lccn.loc. gov/00052514 

■ UT Austin: http: //catalog. lib. utexas.edu/record = 
b5133603~S29 

■ USC: http://library.usc.edu/uhtbin/cgisirsi/ 

x/0/ 0/5?searchdatal=1856407{CKEY) 

Book 3: Langley, Lester D. Simon Bolivar: Venezuelan Rebel, 
American Revolutionary. Lanham: Rowman & Littlefield 
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Publishers, c2009 (OCLC number 256770509). 
Permalinks used: 

■ WorldCat@Auraria: http://aurarialibrary.worldcat 


Table 1. Response Times for Book 1 


Day 

World- 

Cat 

Response time in seconds 

UT 

Skyline LC Austin 

USC 

1 

10.5230 

1.3191 

2.6366 

3.6643 

3.1816 

2 

10.5329 

1.2058 

1 .2588 

3.5089 

4.0855 

3 

10.4948 

1 .2796 

2.5506 

3.4462 

2.8584 

4 

13.2433 

1 .4668 

1 .4071 

3.6368 

3.2750 

5 

10.5834 

1 .3763 

3.6363 

3.3143 

4.6205 

6 

11.2617 

1 .2461 

2.3836 

3.4764 

2.9421 

7 

20.5529 

1 .2791 

3.3990 

3.4349 

3.2563 

8 

12.6071 

1 .31 72 

3.6494 

3.5085 

2.7958 

9 

10.4936 

1.1767 

2.6883 

3.7392 

4.0548 

10 

10.1173 

1 .5679 

1.3661 

3.7634 

3.1165 

11 

9.4755 

1 .1 872 

1 .3535 

3.4504 

3.3764 

12 

12.1935 

1 .3467 

4.7499 

3.2683 

3.4529 

13 

1 1 .7236 

1.2754 

1.5569 

3.1250 

3.1230 

Average 

11.8310 

1.3111 

2.5105 

3.4874 

3.3953 


Table 2. Response Times for Book 2 

Response time in seconds 


Day 

World- 

Cat 

Skyline 

LC 

UT 

Austin 

USC 

1 

10.9524 

1 .4504 

2.5669 

3.4649 

3.2345 

2 

10.5885 

1 .2890 

2.71 30 

3.8244 

3.7859 

3 

10.9267 

1 .3051 

0.2168 

4.01 54 

3.6989 

4 

13.8776 

1 .3052 

1.3149 

4.0293 

3.3358 

5 

10.6495 

1 .3250 

4.5732 

3.5775 

3.2979 

6 

1 1 .8369 

1 .3645 

1 .3605 

3.31 52 

2.9023 

7 

11.3482 

1.2348 

2.3685 

3.4073 

3.5559 

8 

10.7717 

1.2317 

1.3196 

3.5326 

3.3657 

9 

11.1694 

1 .0997 

1 .0433 

2.8096 

2.6839 

10 

19.0694 

1 .6479 

2.5779 

4.3595 

2.6945 

11 

12.0109 

1.1945 

2.5344 

3.0848 

18.5552 

12 

12.6881 

0.7384 

1 .3863 

3.7873 

3.9975 

13 

1 1 .6370 

1 .1 668 

1.2573 

3.3211 

3.6393 

Average 12.1174 

1 .2579 

1.9410 

3.5791 

4.5190 


.org / oclc/256770509 

■ Skyline: http:/ /skyline. cudenver.edu/record=b242 
6349-SO 

> LCOC: http://lccn.loc.gov/2008041868 

■ UT Austin: http: //catalog. lib. utexas.edu/record = 
b7192968~S29 

■ USC: http://library.usc.edu/uhtbin/cgisirsi/ 

x/0/0/5?searchdatal=2755357{CKEY) 

We gathered the data for thirteen days in early 
November 2009, an active period in the middle of the 
semester. For each test, we recorded the response time 
total in seconds. The data is displayed in tables 1-3. We 
searched bibliographic records for three books in five 
library catalogs over thirteen days (3 x 5 x 13) for a total of 
195 response time measurements. The WebSitePulse data 
is calculated to the ten thousandth of a second, and we 
recorded the data exactly as it was presented. 


Table 3. Response Times for Book 3 


Day 

Response time in seconds 

World- UT 

Cat Skyline LC Austin 

USC 

1 

10.8560 

1 .3345 

1 .9055 

3.7001 

2.6903 

2 

10.1936 

1.2671 

1 .8801 

3.5036 

2.7641 

3 

11.0900 

1 .5326 

1 .3983 

3.5983 

3.0025 

4 

10.9030 

1 .4557 

2.0432 

3.6248 

2.9285 

5 

12.3503 

1.5972 

3.5474 

3.6428 

4.5431 

6 

9.1008 

1.1661 

1 .4440 

3.4577 

3.1080 

7 

9.6263 

1.1240 

2.3688 

3.1041 

3.3388 

8 

10.9539 

1.1944 

1 .4941 

2.8968 

3.4224 

9 

11.0001 

1 .2805 

1 .3255 

3.3644 

2.7236 

10 

10.2231 

1 .3778 

1.3131 

3.3863 

3.4885 

11 

10.1358 

1 .2476 

2.3199 

3.4552 

2.9302 

12 

12.0109 

1 .1 945 

2.5344 

3.0848 

18.5552 

13 

11.5881 

1 .2596 

2.5245 

3.8040 

3.8506 

Average 1 0.771 7 

1.3101 

2.0076 

3.4325 

4.4112 


Table 4. Averages 


Response time in seconds 


Book 

World- 

Cat 

Skyline 

LC 

UT 

Austin 

USC 

Book 1 

11.8310 

1.3111 

2.5105 

3.4874 

3.3953 

Book 2 

12.1174 

1.2579 

1.9410 

3.5791 

4.5190 

Book 3 

10.7717 

1.3101 

2.0076 

3.4325 

4.4112 

Average 

1 1 .5734 

1 .2930 

2.1530 

3.4997 

4.1 085 
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Web Page Test results 

URL tested: http://lccn.loc.gov/2009366172 

Test performed from: Seattle, WA 

Test performed at: 2010-02-23 15:02:11 (GMT -08:00) 



Time (seconds) 


I I DNS H Connect H Redirect Q First byte Q Last byte B Error 


Figure 7. Permalink screen shot for the record for the title Hard 
Lessons in the Library of Congress online catalog 

Results 

The data shows the response times for each of the three 
books in each of the five online catalogs over the thirteen- 
day testing period. The raw data was used to calculate 
averages for each book in each of the five online catalogs, 
and then we calculated averages for each of the five online 
catalogs (table 4). The averages show that during the testing 
period, the response time varied between 1.2930 seconds 
for the Skyline library catalog in Denver to 11.5734 seconds 
for WorldCat@Auraria, which has its servers in Ohio. 

University of Colorado Denver: WorldCat@Auraria 

WorldCat@Auraria was routinely over Nielsen's ten- 
second limit, sometimes taking as long as twenty sec- 
onds to load all the files to generate a single webpage. 
As previously discussed, this is due to the high number 
and variety of files that make up a single bibliographic 
record. The files sent also include cover images, but they 
are small and do not add much to the total time. After 
our tests on WorldCat@Auraria were conducted, the 
site removed one of the features on pages for individual 
resources, namely the "similar items" feature. This fea- 
ture was one of the most file-intensive on a typical page, 
and its removal should speed up page loads. However, 
WorldCat@Auraria had the highest average response 
time by far of the five catalogs tested. 


Figure 8. WebSitePulse webpage test bar graph results for 
Library of Congress online catalog record 



Figure 9. WebSitePulse webpage test table results for Library of 
Congress online catalog record 


University of Colorado Denver: Skyline 
(Innovative Interfaces) 

As previously mentioned, the traditional catalog at Auraria 
Library runs on an Innovative Interfaces integrated library 
system (ILS). Testing revealed a missing favicon image file 
that the Web server tries to send each time (item 4 in figure 
3). However, this did not negatively affect the response 
time. The catalog's response time was good, with an aver- 
age of 1.2930 seconds, giving it the fastest average time 
among all the test sites in the testing period. As figure 1 
shows, however. Skyline is a typical legacy catalog that is 
designed for a traditional library environment. 

Library of Congress: Online Catalog (Voyager) 

The average response time for the LCOC was 2.0076 


220 INFORMATION TECHNOLOGY AND LIBRARIES I DECEMBER 2010 




Figure 10. Permalink screen shot for the record for the title 
Hard Lessons in University of Texas at Austin’s library catalog 

seconds. This was the second fastest average among the 
five catalogs tested. While, like Skyline, the bibliographic 
record page is sparsely decorated (figure 7), this pays 
dividends in response time, as there are only two CSS 
files and three GIF files to load after the HTML content 
loads (figure 9). Figure 8 shows that initial connection 
time is the longest factor in load time; however, it is still 
short enough to not have a negative effect. Total file size is 
19.27 KB. As with Skyline, the page itself (figure 7) is not 
particularly end-user friendly to nonlibrarians. 

University of Texas at Austin: Library Catalog 
(Innovative Interfaces) 

UT Austin, like Auraria Library, runs an Innovative 
Interfaces ILS. The library catalog also includes book cover 
images, one of the most attractive NextGen features (figure 
10), and as shown in figure 12, third-party content is used 
to add features and functionality (items 16 and 17). UT 
Austin's catalog uses a Google JavaScript API (item 16 in 
figure 12) and Library Thing's Catalog Enhancement prod- 
uct, which can add book recommendations, tag browsing, 
and alternate editions and translations. Total content size 
for the bibliographic record is considerably larger than 
Skyline and the LCOC at 138.84 KB. It appears as though 
inclusion of cover art nearly doubles the response time; 


Web Page Test results 

URL tested: http://catalog.lib.utexas.edu/record=b7195737~S29 

Test performed from: Seattle, WA 

Test performed at: 2010-02-23 15:06:01 (GMT -08:00) 



Time (seconds) 


I I DNS Connect Redirect First byte I 1 Last byte Error 


Figure 1 1 . WebSitePulse webpage test bar graph results for 
University of Texas at Austin’s library catalog record 



Figure 12. WebSitePulse webpage test table results for 
University of Texas at Austin’s library catalog record 

item 14 is a script, that while hosted on the ILS server, que- 
ries Amazon.com to return cover image art (figures 11-12). 
The average response time for UT Austin's catalog was 
3.4997 seconds. This example demonstrates that response 
times for traditional (i.e., not NextGen) catalogs can be 
slowed down by additional content as well. 

University of Southern California: 

Homer (SirsiDynix) 

The average response time for USC's Homer catalog 
was 4.1085 seconds, making it the second slowest after 
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Web Page Test results 

URL tested: http://library.usc.edu/uhtbin/cgisirsi/x/0/0/5?searchdatal=2770895{CKEY} 

Test performed from: Seattle, WA 

Test performed at: 2010-02-23 15:15:12 (GMT -08:00) 



Time (seconds) 


I I DNS m Connect d Redirect |^] First byte [^] Last byte [H| Error 


Figure 14. WebSitePulse webpage test bar graph results for 
Homer, the University of Southern California’s catalog 



Hom«r i Link st Unfvervty of Souihtrn Cilifomli 

i « » Hwn</IIBf»iyu*<-»aiifumBHVrfliuru;i<jo/0/S?twclvm»l-J770>9S[CKtY) C KQ.' 



Figure 13. Permalink screen shot for the record for the title 
Hard Lessons in Homer, the University of Southern California’s 
catalog 

WorldCat@Auraria, and the slowest among the tradi- 
tional catalogs. This SirsiDynix catalog appears to take 
a longer time than the other brands of catalogs to make 
the initial connection to the ILS; this accounts for much 
of the slowness (see figures 14 and 15). Once the initial 
connection is made, however, the remaining content 
loads very quickly, with one exception: item 13 (see fig- 
ure 15), which is a connection to the third-party provider 
Syndetic Solutions, which provides cover art, a summary, 
an author biography, and a table of contents. While the 
display of this content is attractive and well-integrated 
to the catalog (figure 13), it adds 1.2 seconds to the total 
response time. Also, as shown in item 14 and 15, USC's 
Homer uses the AddThis service to add bookmarking 
enhancements to the catalog. Total combined file size is 
148.47 KB, with the bulk of the file size (80 KB) coming 
from the initial connection (item 1 in figure 15). 

Conclusion 

An eye-catching interface and valuable content are lost 
on the end user if he or she moves on before a search is 


Figure 15. WebSitePulse webpage test table results for Homer, 
the University of Southern California’s catalog 

completed. Added functionality and features in library 
search tools are valuable, but there is a tipping point 
when these features slow down a product's response 
time to where users find the search tool too slow or 
unreliable. Based on the findings of this study, we recom- 
mend that libraries adopt Web response time standards, 
such as those set forth by Nielsen, for evaluating vendor 
search products and creating in-house search products. 
Commercial tools like WebSitePulse make this type of 
data collection simple and easy. Testing should be con- 
ducted for an extended period of time, preferably during 
a peak period — i.e., during a busy time of the semes- 
ter for academic libraries. We further recommend that 
reviews of electronic resources add response time as an 
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evaluation criterion. Additional research about response 
time as defined in this study might look at other search 
tools, to include article databases, and especially other 
metasearch products that collect and aggregate search 
results from several remote sources. Further studies with 
more of a technological focus could include discussions 
of optimizing data delivery methods — again, in the case 
of metasearch tools from multiple remote sources — to 
reduce response time. Finally, product designers should 
pay close attention to response time when designing 
information retrieval products that libraries purchase. 
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